Wavefront sensing from retina-reflected light

ABSTRACT

An eye is illuminated with illumination light. A wavefront image of retina-reflected light is generated and an accommodative eye state value is determined based at least in part on the wavefront image.

CROSS REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. Non-Provisional applicationSer. No. 16/917,893 filed Jun. 30, 2020, which claims the benefit ofU.S. Provisional Application No. 62/928,948 filed Oct. 31, 2019. U.S.Non-Provisional application Ser. No. 16/917,893, and U.S. ProvisionalApplication No. 62/928,948 are expressly incorporated herein byreference in their entirety.

BACKGROUND INFORMATION

In a variety of different optical contexts, the ability to measure orsense a light wavefront is useful. Head mounted displays (HMOs) presentvirtual images to users of the HMD. In some contexts, it is advantageousfor the HMD to determine the location of the eye of the user and/ordetermine where the eyes of the user are focusing. However, conventionalmethods used in HMDs and other optical systems for determining where aneye is focusing can be inaccurate, especially across age demographics

BRIEF DESCRIPTION OF THE DRAWINGS

Non-limiting and non-exhaustive embodiments of the invention aredescribed with reference to the following figures, wherein likereference numerals refer to like parts throughout the various viewsunless otherwise specified.

FIG. 1 illustrates an example HMD that may include infrared in-fieldilluminators and a combiner for redirecting retina-reflected infraredlight to a wavefront sensor, in accordance with aspects of thedisclosure.

FIG. 2 is a top view of an example near-eye optical element thatincludes an illumination layer, a combiner layer, and a display layer,in accordance with aspects of the disclosure.

FIG. 3 illustrates a front view of an eye through an exampleillumination layer, in accordance with aspects of the disclosure.

FIG. 4 illustrates an example optical path of infrared illuminationlight and retina-reflected infrared light, in accordance with aspects ofthe disclosure.

FIG. 5 illustrates an example infrared in-field illuminator including alight source and an example beam-forming element, in accordance withaspects of the disclosure.

FIGS. 6A-6C illustrate an eye in different positions with respect to anarray of infrared in-field illuminators and an example combiner, inaccordance with aspects of the disclosure.

FIG. 7 illustrates a wavefront imaging system that may be utilized in anear-eye optical system, in accordance with aspects of the disclosure.

FIG. 8 illustrates a flow chart for generating an accommodative eyestate value, in accordance with aspects of the disclosure.

FIG. 9 is a block diagram illustration of a lenslet array focusing aplanar wavefront of retina-reflected infrared light onto an image sensoras beam spots, in accordance with aspects of the disclosure.

FIG. 10 is a block diagram illustration of a lenslet array focusing aconverging wavefront of reflected infrared light onto an image sensor asbeam spots, in accordance with aspects of the disclosure.

FIG. 11 is a block diagram illustration of a lenslet array focusing adiverging wavefront of reflected infrared light onto an image sensor asbeam spots, in accordance with aspects of the disclosure.

DETAILED DESCRIPTION

Embodiments of wavefront sensing with in-field illuminators aredescribed herein. In the following description, numerous specificdetails are set forth to provide a thorough understanding of theembodiments. One skilled in the relevant art will recognize, however,that the techniques described herein can be practiced without one ormore of the specific details, or with other methods, components,materials, etc. In other instances, well-known structures, materials, oroperations are not shown or described in detail to avoid obscuringcertain aspects.

Reference throughout this specification to “one embodiment” or “anembodiment” means that a particular feature, structure, orcharacteristic described in connection with the embodiment is includedin at least one embodiment of the present invention. Thus, theappearances of the phrases “in one embodiment” or “in an embodiment” invarious places throughout this specification are not necessarily allreferring to the same embodiment. Furthermore, the particular features,structures, or characteristics may be combined in any suitable manner inone or more embodiments.

Embodiments of an apparatus, system, and method for wavefront sensingdescribed in this disclosure are capable of capturing a wavefront imageof infrared light propagating through the lens of an eye. By determiningthe converging or diverging attributes of the wavefront, anaccommodative state of the eye can be determined. Conventionally,Vergence-Accommodation Conflict (VAC) is used as a surrogate toapproximate the accommodative state of the eye. For example, when twoeyes are narrowed the eyes are likely focused to a near-field object(e.g. a book held close) whereas two eyes that are looking straightahead are likely focused near infinity (e.g. a mountain in thedistance). However, VAC only approximates the accommodative state of theeye. Furthermore, the accommodative response of the eye varies overdifferent age groups. For example, individuals under approximately age45 may accommodate freely while older individuals may have limitedaccommodation response. For these reasons, it would be advantageous tomeasure an accommodative state of the eye rather than approximating theaccommodative state based on vergence.

Embodiments of the disclosure provide a way to measure an accommodativestate of the eye in real time or pseudo real-time. To determine theaccommodative state of the eye, an infrared wavefront that haspropagated through the lens of the eye is measured by a wavefrontsensor. A wavefront image captured by wavefront sensor is analyzed fordivergence or convergence to determine the accommodative state of theeye and a virtual image presented to the eye(s) may be adjusted based onthe determined accommodative state of the eye. An array of infraredin-field illuminators or a photonic integrated circuit (PIC), forexample, may illuminate the eye with infrared illumination light and acombiner is utilized to redirect an infrared wavefront (that propagatedthrough the eye lens and is exiting the pupil) to the wavefront sensor.The infrared in-field illuminators may be configured to emit infraredillumination light that is collimated or near-collimated to a center ofrotation of an eye. These and other embodiments are described in moredetail in connections with FIGS. 1-11 .

FIG. 1 illustrates an example HMD 100, in accordance with aspects of thepresent disclosure. The illustrated example of HMD 100 is shown asincluding a frame 102, temple arms 104A and 104B, and near-eye opticalelements 110A and 110B. Wavefront sensors 108A and 108B are shown ascoupled to temple arms 104A and 104B, respectively. FIG. 1 alsoillustrates an exploded view of an example of near-eye optical element110A. Near-eye optical element 110A is shown as including an opticallytransparent layer 120A, an illumination layer 130A, an optical combinerlayer 140A, and a display layer 150A. Display layer 150A may include awaveguide 158 that is configured to direct virtual images to an eye of auser of HMD 100.

Illumination layer 130A is shown as including a plurality of in-fieldilluminators 126. In-field illuminators 126 are described as “in-field”because they are in a field of view (FOV) of a user of the HMD 100.In-field illuminators 126 may be in a same FOV that a user views adisplay of the HMD, in an embodiment. In-field illuminators 126 may bein a same FOV that a user views an external environment of the HMD 100via scene light 191 propagating through near-eye optical elements 110.While in-field illuminators 126 may introduce minor occlusions into thenear-eye optical element 110A, the in-field illuminators 126, as well astheir corresponding electrical routing may be so small as to beunnoticeable or insignificant to a wearer of HMD 100. Additionally, anyocclusion from in-field illuminators 126 will be placed so close to theeye as to be unfocusable by the human eye and therefore assist in thein-field illuminators 126 being not noticeable or insignificant. In someembodiments, each in-field illuminator 126 has a footprint (or size)that is less than about 200×200 microns. When HMD 100 is being worn by auser, the in-field illuminators 126 may be disposed between 10 mm and 30mm from the eye. In some embodiments, the in-field illuminators 126 maybe placed between 15 mm and 25 mm from the eye of a user. The in-fieldilluminators 126 may be infrared in-field illuminators 126 configured toemit infrared illumination light for eye-tracking purposes, for example.

In some embodiments (not illustrated), a photonic integrated circuit(PIC) may be implemented instead of in-field illuminators 126 to achievea similar function as in-field illuminators 126. For example,outcoupling elements may be positioned similarly to theinfield-illuminators 126 and the outcoupling elements may be providedinfrared light by transparent waveguides. Light sources located at theedge of a frame of the HMD may provide the infrared light into thetransparent waveguides, for example. The outcoupling elements thenredirect the infrared light provided by the waveguides to illuminate aneyeward region. The outcoupling elements may have diffractive orrefractive features to facilitate beam-shaping of the infrared lightreceived from the waveguides. Other techniques (not necessarilyconsidered to be PICs) may also be implemented to achieve a similarillumination function as described with respect to in-field illuminators126. In a VR HMD context, wavefront sensor(s) 108 of this disclosure mayalso be disposed in numerous places in the VR HMD besides a templateposition, as illustrated in FIG. 1 .

As shown in FIG. 1 , frame 102 is coupled to temple arms 104A and 104Bfor securing the HMD 100 to the head of a user. Example HMD 100 may alsoinclude supporting hardware incorporated into the frame 102 and/ortemple arms 104A and 104B. The hardware of HMD 100 may include any ofprocessing logic, wired and/or wireless data interface for sending andreceiving data, graphic processors, and one or more memories for storingdata and computer-executable instructions. In one example, HMD 100 maybe configured to receive wired power and/or may be configured to bepowered by one or more batteries. In addition, HMD 100 may be configuredto receive wired and/or wireless data including video data.

FIG. 1 illustrates near-eye optical elements 110A and 110B that areconfigured to be mounted to the frame 102. In some examples, near-eyeoptical elements 110A and 110B may appear transparent to the user tofacilitate augmented reality or mixed reality such that the user canview visible scene light from the environment while also receivingdisplay light directed to their eye(s) by way of display layer 150A. Infurther examples, some or all of near-eye optical elements 110A and 110Bmay be incorporated into a virtual reality headset where the transparentnature of the near-eye optical elements 110A and 110B allows the user toview an electronic display (e.g., a liquid crystal display (LCD), anorganic light emitting diode (OLED) display, a micro-LED display, etc.)incorporated in the virtual reality headset.

As shown in FIG. 1 , illumination layer 130A includes a plurality ofin-field illuminators 126. Each in-field illuminator 126 may be disposedon a transparent substrate and may be configured to emit light towardsan eyeward side 109 of the near-eye optical element 110A. In someaspects of the disclosure, the in-field illuminators 126 are configuredto emit near infrared light (e.g. 750 nm-1.5 μm). Each in-fieldilluminator 126 may be a micro light emitting diode (micro-LED), an edgeemitting LED, a vertical cavity surface emitting laser (VCSEL) diode, ora Superluminescent diode (SLED).

As mentioned above, the in-field illuminators 126 of the illuminationlayer 130A may be configured to emit infrared illumination light towardsthe eyeward side 109 of the near-eye optical element 110A to illuminatethe eye of a user. The near-eye optical element 110A is shown asincluding optical combiner layer 140A where the optical combiner layer140A is disposed between the illumination layer 130A and a backside 111of the near-eye optical element 110A. In some aspects, the opticalcombiner 140A is configured to receive retina-reflected infrared lightthat is reflected by retina of the eye of the user and to direct theretina-reflected infrared light towards the wavefront sensor 108A. Thewavefront sensor(s) 108 may be located in different positions than thepositions illustrated. In some aspects, the optical combiner 140A istransmissive to visible light, such as scene light 191 incident on thebackside 111 of the near-eye optical element 110A. In some examples, theoptical combiner 140A may be configured as a volume hologram and/or mayinclude one or more Bragg gratings for directing the retina-reflectedinfrared light towards the wavefront sensor 108A. In some examples, theoptical combiner 140A includes a polarization-selective volume hologram(a.k.a. polarized volume hologram) that diffracts (in reflection) aparticular polarization orientation of incident light while passingother polarization orientations.

Display layer 150A may include one or more other optical elementsdepending on the design of the HMD 100. For example, display layer 150Amay include a waveguide 158 to direct display light generated by anelectronic display to the eye of the user. In some implementations, atleast a portion of the electronic display is included in the frame 102of the HMD 100. The electronic display may include an LCD, an organiclight emitting diode (OLED) display, micro-LED display, pico-projector,or liquid crystal on silicon (LCOS) display for generating the displaylight.

Optically transparent layer 120A is shown as being disposed between theillumination layer 130A and the eyeward side 109 of the near-eye opticalelement 110A. The optically transparent layer 120A may receive theinfrared illumination light emitted by the illumination layer 130A andpass the infrared illumination light to illuminate the eye of the user.As mentioned above, the optically transparent layer 120A may also betransparent to visible light, such as scene light 191 received from theenvironment and/or display light received from the display layer 150A.In some examples, the optically transparent layer 120A has a curvaturefor focusing light (e.g., display light and/or scene light) to the eyeof the user. Thus, the optically transparent layer 120A may, in someexamples, may be referred to as a lens. In some aspects, the opticallytransparent layer 120A has a thickness and/or curvature that correspondsto the specifications of a user. In other words, the opticallytransparent layer 120A may be a prescription lens. However, in otherexamples, the optically transparent layer 120A may be a non-prescriptionlens.

While FIG. 1 illustrates an HMD 100 configured for augmented reality(AR) or mixed reality (MR) contexts, the disclosed embodiments may alsobe used in other implementations of an HMD. For example, theillumination layers of this disclosure may be disposed close to adisplay plane of a display of a virtual reality (VR) or prior to afocusing lens of a VR HMD where the focusing lens is disposed betweenthe illumination layer and the display and the focusing lens focusesdisplay light from the display for an eye of a wearer of the VR HMD.

FIG. 2 is a top view of an example near-eye optical element 210 thatincludes an illumination layer 230, a combiner layer 240, and a displaylayer 250. A transparent layer (not illustrated) may optionally beincluded between illumination layer 230 and eye 202, in someembodiments. A plurality of infrared in-field illuminators 237 emitinfrared illumination light 239 to an eyebox area to illuminate eye 202.Plane 206 illustrates a two-dimensional pupil plane 206 in the eyeboxarea being normal to the curvature of the eye 202 at the center of pupil203. FIG. 2 illustrates example array of infrared in-field illuminators237A-237E. Each infrared in-field illuminator 237 in the array isconfigured to emit infrared illumination light 239 to a center ofrotation 241 of eye 202. The different infrared in-field illuminators237 may direct infrared illumination light 239 to the center of rotationof eye 202 at different angles depending on the position of the infraredin-field illuminator with respect to eye 202. For example, infraredin-field illuminators 237A and 237E may include beam-forming elementsthat direct the infrared illumination light to eye 202 at steeper anglescompared to infrared illuminator 237C directing infrared illuminationlight 239 to eye 202 at an angle closer to normal. The center ofrotation 241 of eye 202 remains at a substantially same position withrespect to illuminators 237 even over a large range of gaze angles ofeye 202.

As described above, infrared in-field illuminators 237 may be VCSELs orSLEDs, and consequently infrared illumination light 239 may benarrow-band infrared illumination light (e.g. linewidth of 1-10 nm). Theinfrared illumination light 239 may be collimated or near-collimated sothat at least a portion of the infrared illumination light 239 willpropagate through pupil 203 of eye 202, reflect of off retina 208 andexit eye 202 through pupil 203 as retina-reflected infrared light. Aswill be described in greater detail below, the retina-reflected infraredlight may be received by combiner optical element 240 and redirected towavefront sensor 108A to generate a wavefront image. As described above,alternative illumination layer implementations that utilize outcouplingelements, waveguides, and/or planar waveguides that achieve a similarfunction as infrared in-field illuminators 237 may also be utilized togenerate infrared illumination light 239 that is collimated ornear-collimated.

Wavefront sensor 108A is configured to capture wavefront images that maybe utilized to determine an accommodative eye state value of eye 202,for example. Wavefront sensor 108 may include an infrared bandpassfilter to pass the wavelength of the infrared illumination light 239emitted by the infrared illuminators and block other light from becomingincident on an image sensor of wavefront sensor 108A, in someembodiments.

FIG. 2 shows that scene light 191 (visible light) from the externalenvironment may propagate through display layer 250, combiner layer 240,and illumination layer 230 to become incident on eye 202 so that a usercan view the scene of an external environment. FIG. 2 also shows thatdisplay layer 250 may generate or redirect display light 293 to presentvirtual images to eye 202. Display light 293 is visible light andpropagates through combiner layer 240 and illumination layer 230 toreach eye 202.

Illumination layer 230 may include a transparent substrate that theinfrared in-field illuminators 237 are disposed on. The infraredin-field illuminators 237 may also be encapsulated in a transparentmaterial 232. Transparent material 232 is configured to transmit visiblelight (e.g. 400 nm-750 nm) and near-infrared light (e.g. 750 nm-1.5 μm).

FIG. 3 illustrates a front view of eye 202 through an exampleillumination layer 330, in accordance with aspects of the disclosure. Inthe illustrated embodiment, illumination layer 330 include twenty-oneinfrared in-field illuminators (337A-337U). In the illustrated example,infrared illuminators 337A-337H may be considered an “inner ring” ofinfrared in-field illuminators 337 while infrared illuminators 337I-337Uare considered an “outer ring” of infrared in-field illuminators 337. Assuch, infrared illuminators 337I-337U may direct their infraredillumination light to eye 202 at a steeper angle than infraredilluminators 337A-337H in the inner ring. An illumination angle of theinfrared illumination light 239 from different in-field infraredilluminators 337 may increase as a distance of a particular infraredin-field illuminators 337 increases from middle region 231 of the arrayof infrared in-field illuminators 337.

FIG. 4 illustrates an example optical path of infrared illuminationlight 439 and retina-reflected infrared light 449, in accordance withaspects of the disclosure. In FIG. 4 , an array of infrared in-fieldilluminators 437 emit infrared illumination light 439 to a center ofrotation of eye 202. Only the infrared illumination light from infraredin-field illuminators 437B is shown for illustration and description ofthe optical path of the infrared illumination light, in FIG. 4 .Portions of infrared illumination light 439 (not illustrated) may notnecessarily propagate through the pupil and may be scattered by the irisor cornea. However, at least a portion of infrared illumination light439 propagates substantially normal to pupil plane 206 of eye 202 andpropagates through the cornea 201, anterior chamber, pupil 209, and lens204 of eye 202 before becoming incident upon the retina 208. A portion(e.g. ˜10% for 850 nm light) of infrared illumination light 439 reflectsoff the retina 208 as retina-reflected infrared light 449. The portionof infrared illumination light 439 that propagates through pupil 209normal to (or at least substantially normal to) pupil plane 206 is thelight that can be reflected back out of pupil 209 after reflecting offof retina 208 rather than being absorbed by the interior of eye 202. InFIG. 4 , retina-reflected infrared light 449 propagates through lens204, pupil 209, and cornea 201 to exit eye 202. Retina-reflectedinfrared light 449 then propagates through illumination layer 430 andencounters combiner optical element 440.

Combiner optical element 440 receives retina-reflected infrared light449 and redirects the retina-reflected infrared light 449 to a wavefrontsensor (e.g. wavefront sensor 108). Combiner optical element 440 mayinclude a polarization-selective volume hologram that reflects a firstpolarization orientation (e.g. right-hand circularly polarized light) ofthe retina-reflected infrared light and passes polarization orientationsthat are other than the first polarization orientation. Combiner opticalelement 440 may also include a folding mirror, hologram or lineardiffractive grating, to redirected retina-reflected infrared light 449,in some embodiments. The combiner optical element 440 passes visiblelight.

FIG. 5 illustrates an example infrared in-field illuminator 537 that maybe utilized as infrared illuminators 126/237/337/447, in accordance withaspects of the disclosure. The example infrared in-field illuminator 537illustrated in FIG. 5 includes an infrared light source 531 having anoutput aperture 536 and a beam-forming element 535 disposed over outputaperture 536. Beam-forming element 535 is configured to direct theinfrared illumination light 539 to a center of rotation of an eye. Inthe illustrated embodiment of FIG. 5 , beam-forming element 535 includesa refractive material 538 and a lens curvature 534 may be formed of therefractive material 538 of the beam-forming element 535. The lenscurvature 534 may assist in directing the infrared illumination light539 to a center of rotation of the eye. The beam-forming elements of theinfrared light sources may be configured to increase an illuminationangle of the infrared illumination light 539 as a distance of aparticular beam-forming element increases from a middle region (e.g.231) of the array of infrared in-field illuminators so that the infraredillumination light 539 from each infrared in-field illuminator 537 isdirected to a center of rotation of the eye.

Substrate 532 is a transparent material. Refractive material 538 ofbeam-forming element 535 may be a high-index material having arefractive index of greater than three. In some embodiments, theillustrated refractive beam-forming element 535 is replaced by, orincludes, a diffractive optical element configured to direct theinfrared illumination light 539 to the eye. In some embodiments,beam-forming element 535 is approximately 30 microns wide.

FIGS. 6A-6C illustrates an eye 202 in different positions with respectto an array of infrared in-field illuminators 437 and an examplecombiner optical element, in accordance with aspects of the disclosureIn FIG. 6A, infrared in-field illuminator 437B and 437C emit infraredillumination light 239 to a center of rotation of eye 202. Infraredillumination light 239 may be collimated light. Other infrared in-fieldilluminators 437 in the array may also emit infrared illumination light239 to a center of rotation of eye 202. At least a portion of theinfrared illumination light 239 propagates through the pupil of eye 202and reflects off of retina 208 and propagates back through (exiting) thepupil as retina-reflected infrared light.

FIG. 6B illustrates infrared in-field illuminators 437A and 437Bemitting infrared illumination light 239 to a center of rotation of eye202 when eye 202 has changed a gaze angle of the eye. The eye 202illustrated in FIG. 6B may be gazing up or gazing to the left, forexample.

FIG. 6C illustrates infrared in-field illuminators 437C and 437Demitting infrared illumination light 239 to a center of rotation of eye202 when eye 202 is positioned at yet another gaze angle. The eye 202illustrated in FIG. 6C may be gazing down or gazing to the right, forexample.

Notably, FIGS. 6A-6C illustrate that even when the gaze angle and/orposition of eye 202 changes, different infrared in-field illuminatorsare still able to direct infrared illumination light substantiallynormal to pupil plane 206 and therefore have the infrared illuminationlight 239 propagate through the pupil, reflect off of retina 208,propagate back through the pupil (as retina-reflected infrared light449, not illustrated) to combiner optical element 440 to be redirectedto a wavefront sensor. In other words, the infrared in-fieldilluminators 437 in the array are spaced apart so that at least aportion of the infrared in-field illuminators 437 will be positioned toilluminate a retina of the eye, through a pupil of the eye, withinfrared illumination light propagating approximately normal to a pupilplane of the eye, over a range of eye positions. The range of eyepositions may include the maximum eye position range that humans arecapable of.

In some embodiments, the infrared in-field illuminators 437 in the arrayare selectively illuminated based on where a given infrared in-fieldilluminator 437 (or group of infrared in-field illuminators 437) arepositioned. The infrared in-field illuminators 437 selected arepositioned to illuminate the eye 202 with infrared illumination light239 that will propagate through the pupil at angle substantially normalto pupil plane 206 so that the combiner optical element 440 can receivea usable signal of retina-reflected infrared light 449 to direct to thewavefront sensor. In some embodiments, the infrared in-fieldilluminators 437 are selectively activated (turned on) based oneye-tracking data collected by a separate eye-tracking system of an HMD.For example, if the eye-tracking system determines that eye 202 islooking up, infrared in-field illuminators 437A and 437B may beselectively activated since they may be best positioned to illuminateeye 202 with infrared illumination light 239 that will be reflected offretina 208, back through the pupil to combiner optical element 440. Or,if the eye-tracking system determines that eye 202 is looking down,infrared in-field illuminators 437C and 437D may be selectivelyactivated since they may be best positioned to illuminate eye 202 withinfrared illumination light 239 that will be reflected off retina 208,back through the pupil to combiner optical element 440.

FIG. 7 illustrates a wavefront imaging system 700 that may be utilizedin an HMD or as a near-eye optical system, in accordance with aspects ofthe disclosure. Wavefront imaging system 700 includes an eye-trackingmodule 747 for determining a position of eye 202. In some embodiments,eye-tracking module 747 includes a camera configured to capture infraredimages of eye 202. Eye-tracking module 747 generates eye-tracking data793 that may include a position of eye 202. For example, eye 202 maychange gaze angles in any combination of up, down, right, and left, andeye-tracking module 747 may provide those gaze angles or eye position ineye-tracking data 793 by analyzing images of eye 202.

Display 790 generates visible display light 799 for presenting a virtualimage to a user of an HMD. Visible display light 799 may propagatethrough a near-eye optical element that includes illumination layer 430and combiner optical element 440 with very little (if any) optical losssince the materials in the near-eye optical element are configured topass visible light and combiner 440 may be configured to diffract aparticular bandwidth of infrared light emitted by infrared in-fieldilluminators. Display 790 may include an OLED, micro-LED, or LCD in avirtual reality context. In an augmented reality or mixed realitycontext, display 790 may include a transparent OLED or an LCOS projectorpaired with a waveguide included in a near-eye optical element of anHMD, for example.

In FIG. 7 , illumination logic 770 is configured to control display 790and drive images onto display 790. Illumination logic 770 is alsoconfigured to receive eye-tracking data 793 generated by eye-trackingmodule 747. Optionally, illumination logic 770 is configured toselectively activate (turn on) individual or groups of infrared in-fieldilluminators in an array of infrared in-field illuminators inillumination layer 430. Illumination logic 770 may selectively activatethe infrared in-field illuminators based on the received eye-trackingdata 793.

FIG. 7 shows that retina-reflected infrared light 749 may include adiverging wavefront 751A, a converging wavefront 751B, or a planarwavefront 751C. The wavefront is directed to wavefront sensor 745 viacombiner optical element 440 so that wavefront sensor 745 can capture awavefront image 750 that may be provided to illumination logic 770.Although the optical paths associated with infrared illumination light239/439 are not illustrated in FIG. 7 , the infrared illumination lightgenerally follows the example optical paths illustrated in FIG. 4 .

Example wavefront sensor 745 includes an image sensor 748, a lensletarray 746, and an optional focusing lens 735. Wavefront sensor 745 maybe arranged as a Shack-Hartmann wavefront sensor. Image sensor 748 maybe included in a camera with additional focusing elements. Image sensor748 may include a complementary metal-oxide semiconductor (CMOS) imagesensor, for example. As described previously, the camera may include aninfrared filter configured to pass the wavelengths of theretina-reflected infrared light and reject other light wavelengths. Thelenslet array 746 is disposed in an optical path between the combineroptical element 440 and image sensor 748, in FIG. 7 . Lenslet array 746may be positioned at a plane that is conjugate to a pupil plane 206 ofeye 202. Although not illustrated, additional optical elements (e.g.mirrors and/or lenses) may be included to properly focus theretina-reflected infrared light 749 to wavefront sensor 745, indifferent arrangements.

Illumination logic 770 may be configured to adjust a virtual imagepresented to the eye 202 of a user in response to determining anaccommodative eye state value based on a wavefront image 750 captured bywavefront sensor 745. Since the accommodative state of the eye can bederived from wavefront image 750, a user's refractive error can bemeasured and corrected for. Display images driven onto display 790 maybe tailored to correct for the user's refractive error.

FIG. 8 illustrates a flow chart for generating an accommodative eyestate value, in accordance with aspects of the disclosure. The order inwhich some or all of the process blocks appear in process 800 should notbe deemed limiting. Rather, one of ordinary skill in the art having thebenefit of the present disclosure will understand that some of theprocess blocks may be executed in a variety of orders not illustrated,or even in parallel. Process 800 may be executed by illumination logic770, for example.

In process block 805, an eye is illuminated by infrared illuminationlight (e.g. infrared illumination light 239) from an array of infraredin-field illuminators where the infrared illumination light from eachinfrared in-field illuminator is directed to a center of rotation of theeye. The infrared illumination light may be collimated ornear-collimated.

In process block 810, a wavefront image (e.g. 750) of retina-reflectedinfrared light (e.g. 649) is generated. The retina-reflected infraredlight is the infrared illumination light (e.g. 639) reflected by aretina and exiting a pupil of the eye. In some embodiments, generatingthe wavefront image includes receiving the retina-reflected infraredlight with a wavefront sensor (e.g. 745) including an image sensor and alenslet array. The lenslet array may be positioned in a plane that isconjugate to a pupil plane of the eye.

In process block 815, an accommodative eye state value is determinedbased at least in part on the wavefront image. In some embodiments,determining the accommodative eye state value includes analyzing aspacing of beam spots of the wavefront image generated by microlenses ofthe lenslet array focusing the retina-reflected infrared light onto theimage sensor.

In an embodiment, process 800 further includes adjusting a virtual imagepresented to the eye by a head mounted display in response todetermining the accommodative eye state value.

FIG. 9 illustrates a block diagram illustration of a lenslet array 947focusing a planar wavefront of retina-reflected infrared light 649 ontoan image sensor 949 as beam spots 948, in accordance with aspects of thedisclosure. In the illustrated block diagram example, lenslet array 947includes a plurality of microlenses 947A-947Y that focus correspondingbeam spots 948A-948Y. For example, microlens 947A focuses infrared lightonto image sensor 949 as beam spot 948A, microlens 947B focuses infraredlight onto image sensor 949 as beam spot 948B . . . and microlens 947Yfocuses infrared light onto image sensor 949 as beam spot 948Y.Microlens 947M is the middle microlens in the example 5×5 array ofmicrolenses in lenslet array 947. FIG. 9 illustrates that whenretina-reflected infrared light 649 is a planar wavefront (e.g.wavefront 751C), each beam spot 948 is axially aligned with an opticalaxis of its corresponding microlens that focuses that particular beamspot 948. Accordingly, each beam spot 948 in the example is equidistant.In other examples, beam spots 948 may not necessarily be equidistancefor incoming planar wavefronts.

FIG. 10 illustrates a block diagram illustration of a lenslet array 947focusing a converging wavefront of retina-reflected infrared light 649onto an image sensor 949 as beam spots 1048, in accordance with anembodiment of the disclosure. FIG. 10 illustrates that whenretina-reflected infrared light 649 is a converging wavefront (e.g.wavefront 751B), beam spots 1048 have converged toward middle beam spot1048M. Accordingly, when the beam spots 1048 are converging, a wavefrontimage that captures beam spots 1048 will indicate that the lens systemof eye 202 is focusing at nearer distances. The closer the beam spots1048 converge, the nearer the distance the eye 202 may be focusing to.

FIG. 11 illustrates a block diagram illustration of a lenslet array 947focusing a diverging wavefront of retina-reflected infrared light 649onto an image sensor 949 as beam spots 1148, in accordance with anembodiment of the disclosure. FIG. 11 illustrates that whenretina-reflected infrared light 649 is a diverging wavefront (e.g.wavefront 751A), beam spots 1148 have diverged away from middle beamspot 1148M. Accordingly, when the beam spots 1148 are diverging, awavefront image that captures beam spots 1148 will indicate that eye 202is focusing at a farther distance. In some cases, wavefront 751A is notdiverging but merely less divergent than wavefront 751C and the beamspots 1148 formed on the wavefront image are also not converging, butrather converging less than beam spots 1048 of FIG. 10 . In this case,the lesser extent of the convergence of beam spots 1148 (compared withthe convergence of beam spots 1048) indicates that the eye 202 isfocusing at a farther distance than the more condensed beam spots 1048.Consequently, a greater condensing of the beam spots from the respectivemicrolenses represents a near-focused accommodative eye state valuewhere the eye is focused at a near distance and a lesser condensing ofthe beam spots from the respective microlenses represents a far-focusedaccommodative eye state value where the eye is focused at a fartherdistance.

Although lenslet array 947 or 746 may not be configured exactly asillustrated in FIGS. 9-11 in all implementations, FIGS. 9-11 illustratehow analysis of the positioning of the beam spots will indicate thediverging or converging nature of the wavefront of retina-reflectedinfrared light 649 as well as the magnitude of the divergence orconvergence. Accordingly, a magnitude and nature of the accommodativestate of the lens system of eye 202 may be determined from a wavefrontimage generated by wavefront sensor 745 by analyzing the spacing of thebeam spots.

An algorithm to determine the accommodative eye state value of an eyemay include detecting bright beam spots with sub-pixel resolutionaccuracy. The pupil of the eye may be segmented based on intensitythresholding or other computer vision or machine learning principles. Ofcourse, distortion of any optics in the optical path between the opticalcombiner element and the wavefront sensor may be accounted for. The rawdata from a wavefront image that includes an array of bright spots overa dark background may be converted to a wavefront map and compared to acalibration metric to determine an offset in a spherical curvature of anincoming wavefront, for example.

Embodiments of the invention may include or be implemented inconjunction with an artificial reality system. Artificial reality is aform of reality that has been adjusted in some manner beforepresentation to a user, which may include, e.g., a virtual reality (VR),an augmented reality (AR), a mixed reality (MR), a hybrid reality, orsome combination and/or derivatives thereof. Artificial reality contentmay include completely generated content or generated content combinedwith captured (e.g., real-world) content. The artificial reality contentmay include video, audio, haptic feedback, or some combination thereof,and any of which may be presented in a single channel or in multiplechannels (such as stereo video that produces a three-dimensional effectto the viewer). Additionally, in some embodiments, artificial realitymay also be associated with applications, products, accessories,services, or some combination thereof, that are used to, e.g., createcontent in an artificial reality and/or are otherwise used in (e.g.,perform activities in) an artificial reality. The artificial realitysystem that provides the artificial reality content may be implementedon various platforms, including a head-mounted display (HMD) connectedto a host computer system, a standalone HMD, a mobile device orcomputing system, or any other hardware platform capable of providingartificial reality content to one or more viewers.

The term “illumination logic” or “processing logic” in this disclosuremay include one or more processors, microprocessors, multi-coreprocessors, Application-specific integrated circuits (ASIC), and/orField Programmable Gate Arrays (FPGAs) to execute operations disclosedherein. In some embodiments, memories (not illustrated) are integratedinto the processing logic to store instructions to execute operationsand/or store data. Processing logic may also include analog or digitalcircuitry to perform the operations in accordance with embodiments ofthe disclosure.

A “memory” or “memories” described in this disclosure may include one ormore volatile or non-volatile memory architectures. The “memory” or“memories” may be removable and non-removable media implemented in anymethod or technology for storage of information such ascomputer-readable instructions, data structures, program modules, orother data. Example memory technologies may include RAM, ROM, EEPROM,flash memory, CD-ROM, digital versatile disks (DVD), high-definitionmultimedia/data storage disks, or other optical storage, magneticcassettes, magnetic tape, magnetic disk storage or other magneticstorage devices, or any other non-transmission medium that can be usedto store information for access by a computing device.

A computing device may include a desktop computer, a laptop computer, atablet, a phablet, a smartphone, a feature phone, a server computer, orotherwise. A server computer may be located remotely in a data center orbe stored locally.

The processes explained above are described in terms of computersoftware and hardware. The techniques described may constitutemachine-executable instructions embodied within a tangible ornon-transitory machine (e.g., computer) readable storage medium, thatwhen executed by a machine will cause the machine to perform theoperations described. Additionally, the processes may be embodied withinhardware, such as an application specific integrated circuit (“ASIC”) orotherwise.

A tangible non-transitory machine-readable storage medium includes anymechanism that provides (i.e., stores) information in a form accessibleby a machine (e.g., a computer, network device, personal digitalassistant, manufacturing tool, any device with a set of one or moreprocessors, etc.). For example, a machine-readable storage mediumincludes recordable/non-recordable media (e.g., read only memory (ROM),random access memory (RAM), magnetic disk storage media, optical storagemedia, flash memory devices, etc.).

The above description of illustrated embodiments of the invention,including what is described in the Abstract, is not intended to beexhaustive or to limit the invention to the precise forms disclosed.While specific embodiments of, and examples for, the invention aredescribed herein for illustrative purposes, various modifications arepossible within the scope of the invention, as those skilled in therelevant art will recognize.

These modifications can be made to the invention in light of the abovedetailed description. The terms used in the following claims should notbe construed to limit the invention to the specific embodimentsdisclosed in the specification. Rather, the scope of the invention is tobe determined entirely by the following claims, which are to beconstrued in accordance with established doctrines of claiminterpretation.

What is claimed is:
 1. A head mounted display (HMD) comprising: awavefront sensor; an illumination layer configured to illuminate aneyebox area with illumination light; and an optical element configuredto receive retina-reflected light and redirect the retina-reflectedlight to the wavefront sensor, wherein the retina-reflected light is theillumination light reflected by a retina of an eye.
 2. The HMD of claim1, wherein the illumination layer includes an array of infrared in-fieldilluminators configured to be disposed between 10 mm and 35 mm from theeye when a user of the HMD is utilizing the HMD.
 3. The HMD of claim 2,wherein an individual infrared in-field illuminator has a footprint ofless than 200 microns×200 microns.
 4. A near-eye optical systemcomprising: a wavefront sensor; an array of illuminators configured toilluminate an eyebox area with illumination light; and a combineroptical element configured to receive retina-reflected light andredirect the retina-reflected light to the wavefront sensor, wherein theretina-reflected light is the illumination light reflected by a retina.5. The near-eye optical system of claim 4, wherein each of theilluminators in the array includes: a light source emitting theillumination light; and a beam-forming element configured to direct theillumination light toward a center of rotation of the eye.
 6. Thenear-eye optical system of claim 5, wherein the beam-forming elements ofthe light sources are configured to increase an illumination angle ofthe illumination light as a distance of a particular beam-formingelement increases from a middle region of the array of illuminators. 7.The near-eye optical system of claim 4, wherein the illuminators includeat least one of a micro light emitting diode (micro-LED), an edgeemitting LED, a vertical cavity surface emitting laser (VCSEL) diode, ora Superluminescent diode (SLED).
 8. The near-eye optical system of claim4 further comprising: a transparent substrate, wherein the array ofilluminators is disposed on the transparent substrate, and wherein thetransparent substrate is positioned to pass the retina-reflected lightthrough the transparent substrate to the combiner optical element, thecombiner optical element configured to redirect the retina-reflectedlight back through the transparent substrate toward the wavefrontsensor.
 9. The near-eye optical system of claim 4 further comprising:illumination logic configured to selectively activate individualilluminators in the array of illuminators.
 10. The near-eye opticalsystem of claim 4, wherein the wavefront sensor includes: a cameraincluding an image sensor; and a lenslet array disposed in an opticalpath between the combiner optical element and the image sensor, whereinmicrolenses of the lenslet array focus the retina-reflected light ontothe image sensor.
 11. The near-eye optical system of claim 10, whereinthe lenslet array is positioned at a plane that is conjugate to a pupilplane of an eye.
 12. The near-eye optical system of claim 10, whereinthe camera includes an infrared filter configured to pass theillumination light and reject other light wavelengths.
 13. The near-eyeoptical system of claim 4, wherein the combiner optical element includesa polarization-selective volume hologram that reflects a firstpolarization orientation of the retina-reflected light and passespolarization orientations that are other than the first polarizationorientation, and wherein the combiner optical element passes visiblelight.
 14. The near-eye optical system of claim 4, wherein theilluminators in the array are spaced apart so that at least a portion ofthe illuminators will be positioned to illuminate a retina of the eye,through a pupil of the eye, over a range of eye positions.
 15. A methodcomprising: illuminating an eye with illumination light; generating awavefront image of retina-reflected light, wherein the retina-reflectedlight is the illumination light reflected by a retina of the eye; anddetermining an accommodative eye state value based at least in part onthe wavefront image.
 16. The method of claim 15, wherein generating thewavefront image includes receiving the retina-reflected light with awavefront sensor including an image sensor and a lenslet array.
 17. Themethod of claim 16, wherein the lenslet array is positioned at a planethat is conjugate to a pupil plane of the eye.
 18. The method of claim16, wherein determining the accommodative eye state value includesanalyzing a spacing of beam spots of the wavefront image generated bymicrolenses of the lenslet array focusing the retina-reflected lightonto the image sensor.
 19. The method of claim 15 further comprising:adjusting a virtual image presented to the eye by a head mounted displayin response to the accommodative eye state value.
 20. The method ofclaim 15, wherein the illumination light is collimated ornear-collimated.